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1.  Introduction 


It  is  common  in  science  and  engineering  to  formulate 
mathematical  models  to  describe  complex  systems.  In  science 
such  models  mathematically  describe  the  behavior  of  systems 
governed  by  certain  laws  and  hypotheses.  A  "law"  in  this 
context  is  some  generally  applicable  statement,  such  as 
conservation  of  momentum,  that  is  universally  accepted  as 
true.  A  hypothesis  is  an  assumption,  which  we  wish  to  test. 

A  model  in  this  case  is  a  mathematical  solution  of  a  set  of 
equations  which  represent  the  underlying  laws  and  hypotheses, 
which  are  often  collectively  referred  to  as  a  theory.  It  can 
then  be  used  to  predict  future  behavior  of  such  a  system. 
Comparison  of  such  predictions  with  observations  test  the 
validity  of  the  model,  in  particular  the  underlying 
assumptions.  An  example  is  NCAR ' s  Thermospheric  General 
Circulation  Model  (TGCM)(1>.  This  model  is  a  numerical 
solution  of  a  set  of  equations  expressing  conservation  of 
mass,  momentum,  and  energy,  in  the  thermosphere, 
incorporating  assumptions  concerning,  for  example,  the 
spatial  and  temporal  distributions  of  energy  and  momentum 
sources.  Tests  of  the  theory  (underlying  laws  and 
assumptions)  generally  consists  of  "running"  the  model  for  a 
few  test  cases  for  which  relevant  data  are  available.  Based 
on  a  comparison  of  results  with  the  data,  the  theory  is 
revised  and  rerun.  This  procedure  hopefully  results  in 
improved  understanding  of  the  dynamics  of  the  complex  systems 
that  the  theory  attempts  to  describe. 

In  this  report,  however,  we  will  be  concerned  with  another 
type  of  model,  for  example  the  MSIS  86  thermospheric  model(2>, 
which  is  used  to  obtain  rapid,  yet  hopefully  sufficiently 
accurate,  estimates  of  key  parameters  which  affect  the  design 
and/or  operation  of  a  man-made  or  natural  system.  These 
parameters  may  be  environmental,  primarily  describing  a 
system  which  interacts  with  the  man-made  system,  or  they  may 
relate  to  the  system  itself.  Thus,  for  example,  the  density 
of  the  atmosphere  at  the  location  of  a  satellite  is  an 
important  environmental  parameter,  since  the  atmosphere 
exerts  on  the  satellite  a  drag  force  which  is  proportional  to 
the  density.  The  predictions  of  such  models  are  used  in  the 
design  of  a  system,  or  to  estimate  the  behavior  of  the 
system.  For  example,  in  tracking  a  satellite,  it  is  usually 
necessary  to  predict  its  orbit  over  a  period  of  time,  in 
order  to  provide  look  directions  for  the  tracking  equipment. 
This  requires  an  estimate  of  the  atmospheric  density  for  the 
prediction  period. 

For  some  such  parameters  a  copious  amount  of  data  is 
available.  Thus  empirical  models  are  set  up  with  a  number  of 
adjustable  parameters  whose  values  are  determined  in  least 
squares  fits  to  the  data.  These  models  may  incorporate  the 
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dynamics  of  the  system,  as  do  the  scientific  models,  but 
often  in  some  grossly  simplified  form,  to  permit  ease  of 
everyday  computational  use.  We  then  become  concerned  with 
the  accuracy  of  these  models  in  predicting  the  behavior  they 
describe. 


1 . 1  Definition  of  Accuracy 

A  dictionary  definition  of  accuracy  might  be  the  quality  of 
being  errorless.  Since  such  a  quality  cannot  be 
realistically  expected  of  models,  we  need  a  more  practical 
definition.  We  therefore  prefer  the  phrase  "smallness  of 
error".  Thus  a  model  is  deemed  accurate  if  the  error  made  in 
using  it  is  small.  This  gets  us  to  the  question  of  just  what 
do  we  mean  by  the  term  "error". 

For  any  single  application  of  the  model,  its  error  may  be 
simply  stated  as  the  difference  between  the  true  value  of  the 
quantity  that  the  model  is  predicting,  and  the  value 
predicted  by  the  model.  This  is  usually  called  the  absolute 
error.  In  some  cases,  for  example  atmospheric  density,  it  is 
often  preferable  to  use  the  relative  error,  which  is  the 
ratio  of  the  absolute  error  to  the  the  quantity  predicted  by 
the  model.  Relative  error  is  usually  preferable  in  cases 
where  the  quantity  in  question  varies  by  an  order  of 
magnitude  or  more  over  the  data  set,  possibly  causing  the 
absolute  errors  to  be  smallest  when  the  quantity  being 
measured  is  smallest,  thus  giving  us  an  unrealistic 
assessment  of  the  accuracy  of  the  model  for  these  cases.  For 
atmospheric  density,  it  has  been  customary  to  use  the  ratio 
of  the  "true"  density  (as  given  by  a  measurement)  to  the 
model  density,  which  is  in  fact  the  relative  error  plus  1. 

The  error  (relative  or  absolute)  of  a  model  is  naturally  not 
the  same  for  all  cases.  Ideally  we  would  therefore  need  to 
specify  the  error  for  each  possible  instance  in  which  the 
model  is  applied.  Since  this,  of  course,  is  also  impossible, 
we  therefore  seek  the  probability  density  function  which 
represents  the  fact  that  certain  values  of  the  error  occur 
more  frequently  than  others.  The  probability  that  a 
continuous  variable  has  a  value  between  x  and  x+dx  is: 

f  ( x ) dx 

where  f(x)  is  thus  the  probability  density  function  for  the 
variable.  The  function  f(x)  must  be  positive,  and  satisfy 

/f (x)dx  =  1  (1) 

where  the  integration  is  over  the  entire  range  of  x.  This 
simply  states  that  the  probability  of  the  variable  taking  on 
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some  value  within  its  range  is  unity.  From  the  probability 
density  function  we  can  immediately  define  two  commonly  used 
measures  of  error,  the  mean  error 

|x  =  Jxf(x)dx  (2) 

and  the  standard  deviation: 

a  =  [  /(x-^,)2f (x)dx]1/2  (3) 

A  commonly  occurring  probability  density  function  is  the 
Gaussian  or  normal  function,  which,  for  mean  (x  and  standard 
deviation  a  is  given  by: 

fg(x)  =  (2'TTCT2)-1/2exp[-(x-p.)2/(2CT2)]  (4) 


The  probability  density  function  can  in  practice  be  estimated 
by  comparing  the  model  to  the  measurements  in  a  properly 
constructed,  accurate  and  sufficiently  large  data  set.  The 
data  set  is  properly  constructed  if  it  correctly  samples 
cases  similar  to  those  for  which  the  model  user  has  an 
interest,  and  omits  cases  that  are  not  similar.  For  example, 
a  user  who  needs  to  track  a  satellite  which  has  an  orbit  at 
1000  and  2200  hours  local  time  is  interested  in  the  accuracy 
of  the  density  model  just  at  those  two  local  times,  not  at 
other  times.  Therefore  the  optimum  data  set  for  this  user 
might  consist  of  on-board  measurements  by  sun -synchronous 
satellites  at  1000  and  2200. 

The  accuracy  of  the  data  set,  the  differences  between 
measured  and  true  values,  is  another  matter  of  concern.  The 
mean  value  of  these  errors  is  often  called  the  systematic 
error.  Often  we  have  only  a  poor  idea  of  these  errors,  since 
we  cannot  obtain  an  independent  measure  of  them.  For 
instance,  the  analysis  of  on-board  satellite  accelerometer 
drag  measurements  to  yield  the  atmospheric  density  relies  on 
an  accurate  value  for  the  satellite's  drag  coefficient. 

Since  the  drag  coefficient  cannot  in  practice  be  measured 
under  controlled  conditions,  it  must  be  estimated  from  a 
model,  whose  accuracy  in  turn  is  poorly  estimated.  This 
could  result  in  a  roughly  constant,  and  therefore  systematic, 
but  uncertain,  error  in  the  resulting  "measured"  densities. 

From  a  given  data  set  it  is  customary  to  compute  the  sample 
mean  and  standard  deviation: 

M  =  Sxi/N  (5) 

s  =  [2(  Xi~M  )2/  ( N- 1 )  ] 1/2  (6) 

as  estimates  of  the  mean  jx  and  standard  deviation  ct  given  in 
Equations  ( 2 )  and  ( 3 ) . 
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The  data  set  must  be  sufficiently  large  that  the  sampling 
error  in  these  values,  which  varies  as  1/Jn,  is  small  com¬ 
pared  to  the  model  errors  we  are  determining.  If  the  data 
set  meets  the  conditions  of  proper  sampling,  negligible 
measurement  error,  and  large  sample  size,  then  these 
estimates  should  be  sufficiently  accurate  to  meet  practical 
needs.  If  the  only  problem  with  the  data  is  systematic 
error,  then  the  estimate  s  for  the  a  is  not  affected,  but 
the  estimate  M  for  jx  is. 

Another  way  to  specify  accuracy  is  through  the  use  of 
confidence  limits.  Given  a  probability  density  function,  we 
can  compute  the  total  probability,  the  integral  of  the 
probability  density  function,  that  the  error  lies  between  two 
values: 


P(a,b) 


’b 

f ( x )dx 
a 


(7) 


Often  we  wish  to  find  the  bounds  a  and  b  of  an  interval  whose 
center  is  at  the  mean  of  the  distribution.  In  this  report, 
however,  we  show  how  to  define  the  confidence  limits  +D  of  an 
interval  whose  center  is  at  zero  error,  such  that  the 
probability  P(-A,+A)  that  the  error  lies  within  this  interval 
is  at  least  equal  to  some  desired  value.  This  specification 
has  the  advantage  that  it  gives  the  user  the  desired 
"confidence"  that  the  error  magnitude  is  less  than  the  half¬ 
width  A  of  the  interval. 


We  desire  some  simple  parametric  specification,  accompanied 
by  a  table  or  simple  computational  algorithm,  which  would 
allow  the  user  to  easily  derive  confidence  limits  for  any 
specified  probability.  This  specification  must  be  applicable 
in  the  presence  of  uncertain  systematic  error  and  non-zero 
mean  error.  We  will  show  that  if  the  probability  density 
function  is  Gaussian,  with  zero  or  non-zero  mean,  then  a 
single  parameter  nevertheless  often  suffices  to  derive 
desired  confidence  limits.  For  non-Gaussian  probability 
density  functions,  a  modification  of  Chebyshev's  inequality(3) 
will  allow  us  to  derive  from  the  same  parameter  an  upper 
bound  to  the  confidence  interval  width. 


2.  Methods 


We  discuss  some  simple  properties  of  confidence  limits, 
given  a  probability  density  function  which  may  or  may  not  be 
centered  about  zero.  These  properties  are  therefore  useful 
for  the  case  of  non-zero  mean  error  as  well  as  zero  mean. 
They  are  applicable  to  any  probability  density  function,  of 
which  the  Gaussian  is  a  special  case,  which  monotonically 
decreases  with  increasing  distance  from  the  mean. 
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Properties  of  Confidence  Limits 


Let  g(x)  be  a  function  which  decreases  monotonically  with 
increasing  absolute  value  of  its  argument.  Therefore  g  is  an 
even  function  of  its  argument  x.  The  Gaussian  function  for  a 
variable  with  zero  mean  is  a  special  case.  We  define 
p^tf'  (L)  as  the  probability  that  a  variable  whose  probability 
density  is  given  by  g  lies  within  the  limits  ±L-A  (L  t  0): 

r-A 

g(x)dx  = 

L-A 


The  second  equality  expresses  the  fact  that  this  is  also  the 
probability  that  the  variable  whose  probability  density  (pd) 
is  g(x-A),  with  mean  value  A,  lies  within  the  limits  ±L. 

We  now  show  the  following: 

(i)  P  is  an  even  function  of  A: 

P_a<0)  =  PA<0)  (9) 

Applying  the  change  of  variable  s  =  -x  to  the  expression 
immediatly  after  the  first  equals  sign  of  Eq.  (8),  and 
using  the  fact  that  g  is  an  even  function: 


h 


g( x-A )dx 


(8) 


r- l*a 

PA<«>(L)  =  -  g(s)ds 

J  L*A 
»L*A 

=  g(s)ds 

-L*A 

=  P-a(0)(L) 

(ii)  As  a  function  of  A,  P  has  a  single  maximum  at  A=0: 
PA<o>(L)  <  Po<0>(L)  (if  A*0) 


(10) 


The  integral 


fL«A 

g(x)dx  breaks  down  into  the  sum: 

-L*A 


f-L 


g(x)dx  + 


-t*A 


g(  x )dx  + 


-L 


PL*A 

g  ( x )  dx 

L 


The  second  term  is  P0.  The  difference  PA-P0  is  the  sum 
of  the  first  and  third  terms.  We  must  therefore  prove 
that  this  sum  is  negative.  Since  g(x)  is  an  even 
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function,  the  first  term  may  be  written  as: 

'L-A 

g(x)dx 

L 

Applying  the  change  of  variable  x  =  L-s  to  the  first 
term  in  the  sum,  and  x  =  L+s  to  the  other  term,  we 
obtain 


r 

g( L-s )ds  + 


r 

g(L+s)ds 


J  o 


j  o 


For  A  positive,  these  integrals  are  over  positive  s. 

Then  the  absolute  value  of  the  argument  L-s  of  the  first 
integrand  is  less  than  the  absolute  value  of  the  argu¬ 
ment  L+s  of  the  second  integrand.  Therefore  the  first 
integrand  is  larger  than  the  second  (both  positive), 
and  the  sum  is  negative,  as  required  for  our  proof. 

For  negative  A,  the  result  follows  from  property  (i) 
just  shown  above,  the  invariance  of  PA  under  change  of 
sign  of  A. 


Finally,  we  note,  for  the  Gaussian: 

Pa(CT,(L)  =  {erf  [  ( L+A)  /  ( cr42  )]+erf[(L-A)/(aJ2)]}/2  (11) 


3 .  Results  and  Applications 

As  stated  previously,  sometimes  we  are  not  interested  in  the 
deviation  of  the  model  error  about  its  mean,  but  rather  we 
are  interested  in  the  total  error,  or  put  another  way,  the 
deviation  of  the  error  about  zero.  Thus  we  have  to  deal  with 
the  probability  in  a  region  not  centered  about  the  mean.  In 
the  previous  section  we  gave  some  theorems  concerning  this 
situation,  and  in  addition,  Eq.  (11)  for  application  to  a 
biased  Gaussian,  that  is,  a  Gaussian  pd  with  non-zero  mean. 

We  will  now  demonstrate,  for  the  Gaussian,  a  rather 
surprising  result  that  the  probability  for  occurrence  of  the 
error  within  a  given  interval  centered  about  zero  is  often 
approximately  equal  to  that  which  would  be  obtained  for  an 
unbiased  Gaussian  pd  with  standard  deviation  equal  to  the 
root  mean  square  (rms)  of  the  mean  and  the  standard  deviation 
of  the  original  Gaussian  pd.  We  will  next  demonstrate  how 
the  rms  can  be  used  to  obtain  an  upper  bound  for  the 
confidence  limits  for  non-Gaussian  distributions.  These 
results  allow  us  to  use  the  rms  as  a  single  parameter  to 
characterize  the  model  error  in  many  cases. 


6 


3 . 1  Probabilities  for  a  Biased  Gaussian 

Assume  that  a  data  point  is  taken  from  Gaussian  probability 
density  with  standard  deviation  a  and  mean  (bias)  Bcr.  Then 
Table  1  gives  the  probability  for  the  occurrence  of  the  value 
(y)  within  the  interval 


-ctAx  <  y  <  crAx, 


where  Ax  is  the  value  indicated  first  column,  and  B  is  the 
value  indicated  in  the  top  line.  For  example,  if  Ax  =  .5, 
and  B  *  .4,  then  the  probability  of  occurrence  within  the 
interval  ±crAx  is  35.58%.  If  Ax  =  1,  and  B=  0,  then  the 
probability  is  68.27%. 

Assume  that  a  data  point  is  taken  from  a  Gaussian  pd  with 
standard  deviation  a  and  bias  Bcr.  Then  Table  2  gives  the 
probability  of  occurrence  of  the  value  (y)  within  the 
interval 

-[cr2  +  B2cr2]l/2AxN  ^  y  ^  [a2  +  cr2B2]  1/2AxN, 

where  AxK  is  the  value  indicated  in  the  first  column,  and  B 
is  the  value  indicated  in  the  top  line. 

Table  2  shows  that  by  modifying  a  to 

rms  *  [a2  +  <t2B2]1/2  (12) 

the  probability  of  occurrence  within  symmetric  intervals  is 
approximately  independent  of  the  bias  (B)  when  B  is  "small". 
For  example  the  probability  of  occurrence  of  y  within  the 
interval  ±.5rms  is  38.29%  when  the  bias  is  zero,  and  38.11% 
when  the  bias  is  .  4<r. 


3.2  Probabilities  for  Non-Gaussian  Distributions 

Chebyshev's  inequality  states  an  upper  bound  for  the 
probability  that  the  value  of  a  random  variable  deviates  from 
its  mean  by  more  than  a  certain  amount: 

P{|x  -  \l\  >  A}  <  cr2/A2 

where  p.  and  a  are  the  mean  and  standard  deviation  of  the 
value  of  the  random  variable  x,  and  A  is  a  positive  number. 
Brunk(3>  points  out  that  the  proof  of  Chebyshev's  inequality 
applies  equally  well  to  deviations  of  the  random  variable 
from  any  value,  if  the  standard  deviation  a  is  replaced  by 
the  square  root  of  the  mean  square  deviation  of  the  variable 
from  that  value. 


7 


The  proof  uses  a  step  function: 

8(t)  =1,  t  >  A 
=  0,  t  <  A 

It  is  seen  for  z  =  x  -  p,  that: 
z2  £  A20(  |z|  ) 

since,  when  z2  is  less  then  A2,  the  right  hand  side  is  zero, 
and  when  z2  equals  or  exceeds  A2,  the  right  hand  side  is 
equal  to  A2.  Taking  mean  values: 

< z2>  £  <A20 (  | z  j  ) >  =  A2<0(  j z |  )>  =  A2P C  | z |  >  A} 

The  first  equality  follows  from  the  constancy  of  A,  the 
second  from  the  definition  of  the  step  function.  We  see 
therefore  that 

P{  |  z  |  >  A}  £  <z2>/A2 

Thus  far  no  use  has  been  made  of  the  property  that  p.  is  the 
mean  value  of  x.  This  is  done  only  in  the  final  step  of  the 
proof,  substituting  cr2  for  <z2>,  the  mean  squared  value  of  z. 
Thus,  when  considering  the  deviation  from  a  value  other  than 
the  mean,  we  may  replace  a2  in  Chebyshev's  inequality  with 
the  mean  square  deviation  from  the  selected  value.  Then 
Chebyshev's  inequality  may  be  stated  in  the  modified  form: 

P{|x  -  x0|  £  A}  <,  < ( x  -  x0)2>/A2 

for  the  deviation  of  x  from  any  value  x0.  For  our 
application  to  the  deviation  of  errors  about  zero,  the  mean 
square  deviation  is  simply  the  rms  defined  in  Eq.  (12),  with 
p.  =  Bcr,  since 

<x2>  =  <[(x  -  p.)  +  p,]2> 

=  <(x  -  p,)2  +  2p.(x  -  p,)  +  p,2> 

*  <(x  -  p,)2>  +  2p,<(x  -  p,)>  +  p,2 

a  (X2  +  p,2 

We  therefore  finally  state  that 
P{ |x|  £  A}  ^  rms2/A2 

meaning  that  the  probability  that  x  is  outside  the  interval 
[-A,  A]  cannot  exceed  (rms/A)2.  Thus  the  probability,  or 
confidence,  that  the  error  x  is  within  this  interval  is  at 
least  l~(rms/A)2. 


8 


We  may  use  this  result  to  obtain  an  upper  bound  for  the  half 
width  A  of  the  confidence  interval,  such  that  the  probability 
that  the  error  x  is  within  this  interval  is  at  least  a 
desired  value  a: 

A  <  rms/(l-a)1/2  (13) 

We  note  that  the  upper  bound  given  by  the  right  hand  side  of 
Eg.  (13)  for  68.2%  confidence  is  1.75  rms,  compared  to  1.0 
rms  for  Gaussians,  as  indicated  by  Table  2.  For  95% 
confidence  the  upper  bound  given  by  Eq.  (13)  is  4.47  rms, 
compared  to  1.96  rms  for  Gaussians.  Futhermore  we  note  that 
the  interval  half -width  A  from  Eq.  (13)  can  never  be  less 
than  one  rms . 


3.3  Applications  to  Atmospheric  Density  Model  Errors 

AFGL  has  collected  a  substantial  data  base  of  in-situ 
satellite  drag  measurements  in  the  lower  thermosphere (4) .  The 
densities  derived  from  this  data  can  then  be  compared  with 
models  such  as  the  previously  mentioned  TGCM(1)  and  MSIS-86(2) 
model.  With  respect  to  engineering  models  such  as  the  MSIS- 
86,  the  following  statistics  have  routinely  been  computed: 

M  -  £  rj  N  (14) 

s  =  [2  (ri-M)/(N-l)]i/2/M  (15) 

where  r*  is  the  ratio  of  the  ith  measured  density  to  the 
density  predicted  by  the  model  for  that  case,  and  N  is  the 
number  of  measurements.  These  quantities  are  similar  to  the 
sample  estimates,  Eqs.  (2)  and  (3),  for  the  mean  and  standard 
deviation  of  the  model  ratio  r,  except  that  in  the  present 
case  the  sample  estimate  for  the  standard  deviation  has  been 
divided  by  the  mean  ratio  M.  With  these  two  parameters,  the 
model  user  may  imply  that  a  corrected  model  obtained  by 
multiplying  the  original  model  by  M  will  have  relative  errors 
with  estimated  standard  deviation  s.  There  are  at  least  two 
problems  with  this  approach.  One  is  that  the  user  may  need 
to  know  the  error  in  the  model  itself,  rather  than  in  a 
corrected  version.  The  second  problem  is  that  the  data  base 
may  have  systematic  error,  which  introduces  uncertainty  in 
the  mean  M,  although  not  in  the  standard  deviation  s.  The 
following  examples  will  show  how  to  characterize  the  error  in 
the  model  itself  (not  the  corrected  model)  given  that  we  have 
a  sample  estimate  M  of  the  mean  ratio,  such  that  a  sample 
estimate  of  the  mean  relative  error  |x  is  M-l,  where  M  is 
given  by  Eq.  (14),  and  a  sample  estimate  of  the  standard 
deviation  or  given  by  Ms,  where  s  is  given  by  Eq.  (15).  The 
reason  for  using  Ms  instead  of  s  is  that  we  are  interested 
here  in  the  relative  error  of  the  original  model,  not  the 
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relative  error  of  the  corrected  model .  The  factor  M  is 
generally  close  to  one,  unless  the  model  is  very  poor. 


3.3.1  Model  Developed  from  a  Data  Base 

We  assume  that  a  model  has  been  developed  empirically  from  a 
data  base.  This  is  generally  the  case  for  models  such  as  the 
MSIS-86.  This  data  base  is  then  used  to  "evaluate"  the 
model.  The  mean  relative  model  error  is  found  to  be  zero, 
and  the  standard  deviation  is  found  to  be  15%. 

A.  The  data  had  no  systematic  or  random  error.  The  rms  is 
therefore  15%.  If  the  error  distribution  is  Gaussian,  we 
may  then  use  column  1  of  Table  2,  to  find  the  ±  error 
range  for  any  desired  probability.  Simply  find  the  row 
containing  the  desired  probability  in  column  2  and 
multiply  the  number  in  column  1  by  15%.  If  the  distri¬ 
bution  is  strongly  non-Gaussian,  we  may  use  Eq.  (13)  to 
find  an  upper  bound  for  the  error  range. 

B.  We  find  out  later  that  the  measurements  should  have  been 
10%  higher.  Therefore,  using  the  hypothetically  corrected 
data  base  to  evaluate  the  model  obtained  on  the  basis  of 
the  uncorrected  data,  we  would  obtain  a  mean  error  of  10% 
and  a  a  of  (15  x  1.1)%  =  16.5%.  Thus  our  model  error  is 
characterized  by: 

rms ( % )  *  [102  +  16. 52)1'2  =  19.3%. 

C.  We  find  out  later  that,  instead,  there  was  uncertain 
systematic  error  in  the  data,  in  the  range  ±  10%.  Then 
the  total  error  is  the  sum  of  the  measured  and  systematic 
errors.  Assuming  that  the  latter  may  be  characterized  by 
a  probability  density  function  with  zero  mean  and  standard 
deviation  10%,  we  obtain  for  the  total  error: 

rms  -  [102  +  152] 1/2  =  18.0% 

If  the  probability  density  functions  of  both  the  measured 
systematic  errors  are  Gaussian,  then  the  probability 
density  function  of  the  total  error  is  also  Gaussian(3), 
and  we  may  use  Table  2,  as  above,  to  determine  the 
confidence  limits  for  any  specified  probability.  If 
either  distribution  is  non-Gaussian,  we  may  use  Eq.  (13). 

There  is  another  view  of  the  systematic  error  which  is 
sometimes  considered:  the  systematic  error  is  known 
definitely  to  be  within  a  certain  range,  but  its  pd  within 
that  range  is  unknown.  Thus  the  total  error  pd  may  be 
biased  by  some  unknown  amount,  within  this  range.  If  the 
measured  error  distribution  is  Gaussian,  then  according  to 


10 


our  results  of  Section  2.1,  property  (ii),  the  probability 
PA  of  occurrence  within  an  interval  centered  about  zero 
decreases  with  the  increasing  magnitude  of  the  bias  A. 
Therefore  the  worst  case,  within  this  specification,  would 
occur  if  the  systematic  error  had  the  maximum  value,  10%. 
Using  this  worst  case  we  get  the  same  total  rms  as  just 
given  for  the  10%  a  uncertainty  in  the  systematic  error. 
The  same  rms  is  applicable  if  the  measured  error  distri¬ 
bution  is  unspecified  or  strongly  non-Gaussian,  since  the 
modified  Chebyshev's  inequality  applied  to  this  worst  case 
rms  produces  the  worst  case  confidence  interval. 

Another  method  of  handling  the  systematic  error,  when  it 
is  known  to  lie  within  a  definite  finite  range,  is  to  add 
the  maximum  absolute  value  of  the  systematic  error  to  the 
rms  random  measurement  error.  This  is  the  approach 
suggested  by  Cameron(5) .  It  is  clear  that  our  result, 
using  the  rms  of  the  systematic  and  random  errors,  yields 
tighter  confidence  limits. 


3.3.2  Model  Compared  With  Independent  Data  Base 

A.  The  mean  ratio  is  1.0,  the  standard  deviation  is  15%,  and 
there  is  no  systematic  or  random  measurement  error.  Then 
the  rms  is  the  same  as  in  3.3.1  A,  15%. 

B.  Same  as  A,  except  that  the  mean  ratio  is  1.1.  Then, 
following  our  discussion  in  Sections  3.1  and  3.2, 

rms  -  [102  +  152]i/2  *  18.0%. 

C.  Same  as  B,  except  that  the  data  has  uncertain  systematic 
error,  and  therefore  an  uncertainty  in  the  mean  error, 
within  10%.  If  this  uncertainty  is  characterized  by  a 
probability  density  function  with  zero  mean  and  standard 
deviation  10%,  then  the  overall  pd  (measured  +  systematic 
error)  is  characterized  by: 

a  -  [102  +  152] 1/2  =  18.0% 

JJL  *  10% 

For  this  biased  distribution  we  may  again  define  an 
effective  rms 

rms  =  [ct2  +  jjl2]1/2  =  20.6% 

If  if  we  take  the  second  approach,  mentioned  above,  for 
the  systematic  error,  namely  that  we  know  only  that  it  is 
in  the  interval  ±10%,  then,  taking  the  worst  case,  as  we 
did  previously,  we  get  a  distribution  whose  mean  is  the 
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sum  of  the  measured  mean  and  the  largest  possible  bias 
value: 


p,  =  10%  +  10%  =  20% 

The  standard  deviation  is  as  measured,  15%.  Then,  we 
obtain,  for  the  rms: 

rms  =  [202  +  152]1'2  =  25% 


D.  Same  as  C,  except  that  the  data  has  random  error  of  5%. 

In  this  case,  we  know  that  the  measured  a,  estimated  by  s, 
is  the  standard  deviation  for  the  sum  of  two  independent 
unbiased  random  errors:  the  true  model  (random)  error 
(true  value  -  model  value)  and  the  measurement  error 
(measured  value  -  true  value).  Then  it  is  evident  that 
the  true  model  random  error  has  the  standard  deviation 

cr  =  [  152  -  52] 1/2  =  14.1%. 

Then  we  obtain  for  the  rms,  assuming  that  the  systematic 
error  uncertainty  is  charactized  by  10%  standard  deviation 
and  zero  mean: 

rms  =  [14. 12  +  102  +  102]1'2  =  20.0% 

For  the  second  view  of  systematic  error,  assuming  the 
worst  case  value  of  10%: 

rms  =  [14. 12  +  202]1/2  =  24.5% 

For  Gaussian  distributions  we  must  caution  that,  for  the 
conservative  worst  case  systematic  errors  assumed  in  the 
last  two  examples,  the  biases  obtained  were  larger  than 
one  sigma  (1.33a  in  example  C,  and  1.4a  in  example  D). 

For  biases  this  large.  Table  2  indicates  moderate  decrease 
in  the  probability  of  occurrence  within  intervals  smaller 
than  the  rms,  relative  to  the  probabilities  of  occurrence 
in  the  same  intervals  for  zero  bias.  At  one  rms,  the 
probability  has  decreased  from  68.2%  to  approximately 
63%.  However,  it  is  seen  that  this  trend  eventually 
reverses  as  the  interval  increases. 


4.  Discussion  and  Conclusions 


We  have  attempted  to  find  a  minimal  set  of  parameters  to 
describe  model  accuracy.  It  is  hoped  that,  in  most 
situations,  such  information  would  provide  the  potential  user 
with  all  that  he  needs  to  know  to  assess  the  impact,  on  his 
situation,  of  any  error  in  the  model. 
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We  have  approached  the  problem  from  the  standpoint  of 
confidence  limits,  the  bounds  of  an  interval,  centered  about 
zero,  for  which  the  error  has  a  desired  probability  of 
occurring.  This  requires  determination  of  the  probability 
density  function  for  the  error.  We  have  stated  two  useful 
properties  of  a  certain  class  of  biased  probability  density 
functions  which  include  the  Gaussian.  For  the  Gaussian 
itself  we  have  shown  that  this  probability  often  can  be 

•  calculated  from  a  single  parameter,  the  rms  of  the  mean  and 
standard  deviation,  even  when  the  mean  is  non-zero.  Using  a 
modification  of  Chebyshev's  inequality,  we  have  also  found 

•  the  minimum  probability  of  occurrence  for  any  distribution 
of  specified  mean  and  standard  deviation.  This  leads  to 
confidence  limits  considerably  larger  than  those  obtained  for 
Gaussian  distributions. 

Finally  we  have  applied  these  results  to  atmospheric  density 
model  evaluations,  which  are  carried  out  by  comparing  the 
predictions  of  the  model  with  measurements  in  a  large  data 
base.  We  have  considered  contamination  of  the  data  by  random 
and/or  systematic  measurement  errors.  Although 
the  relevant  error  sources  are  often  Gaussian,  this  is 
not  always  the  case.  Marcos,  et.  al.,  have  found  occasional 
departures  from  the  Gaussian  in  their  evaluations*6* .  In 
addition  to  the  mean  and  standard  deviation,  two  additional 
parameters,  called  skewness  and  kurtosis,  were  used  by  them 
to  characterize  these  cases.  If  the  skewness  is  zero,  then 
the  function  is  still  symmetric  about  its  mean,  so  that  we 
may  still  be  able  to  apply  the  two  properties  mentioned  in 
Section  2.  This  would  not  be  the  case  if  the  skewness  is 
non-zero,  for  this  would  indicate  that  the  function  is  not 
symmetric  about  its  mean.  Such  a  function  cannot  therefore 
depend  only  on  the  absolute  value  of  the  difference  between 
its  argument  and  the  mean,  and  therefore  cannot  fall  into  the 
class  of  functions  discussed  there.  However  if  these  depar¬ 
tures  are  small,  and  we  need  to  decide  just  what  "small"  is, 
these  results  may  still  be  applicable.  Although  Chebyshev's 
inequality  could  be  applied  to  these  cases,  it  is  evident 
that  confidence  limits  considerably  closer  to  those  for 
Gaussian  should  be  obtainable.  Therefore  we  should  look  for 
methods  to  extend  to  near-Gaussians  the  results  obtained  here 
for  Gaussians. 
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oa<ooinnp*NnnnMOnnar-^a«naininoN--oqNaanin«'-ii)onui.aaoiaaooa 

OMUNaaonwnfltinn'-MiocoN.hNtiin.sNia'-nnvhaaaaaAaaaoiQOOQ 

«<-«*««»a«ra^®«r»-»f»onaao  —  n<raa^r>aaaaaaaaaaaaaaaaaaoooo 
--MMnn«<rmua«i>-r.r«aaa«aaaaaaaaaaaaaaaaaaaaaaaaaaoaoo 

^ 

o«NO-»»o«'«»«ni.t«r;a«MO#hn#h«o#Mii(it-#onBs«««»oooooo 

o«rtaNT«nB>-O0nanBt>nahTaNnnoa<-«a-nnai.aaanoaaaaaoooaoo 

«asc(tatainptaN««N«Na'N<tmi(ihr.aooaaaaaaaaaaaaaaapopoop 

-^Mnf»4WMWWWi«>*r-aaaaaaaaaaaaaaa5aaaaaaaaaaaa5So5ao5 

aanN-««Ntrr.onntt  <«rtnnnart<tat>nMi-nonno«onn«aaa»ooooooeo 
on«aaMOMO'rm-n-nnNmnc«r.aat^'ratra-nwwi-aaaaaaaaoaoooooooo 

N«~’cDMHa<ra?AnNonttadNr>«m<0h.acoioaaaaaaaaaaaaaaaQoopooo0 

»NHntr<vwa(oar.Naaaaaaaaaaaaaaaaaaaaaaaaaaaaaoooooooo 


s 


-»«09«<r-»'--««-«n«!i(t»N«'Oiiror(aoBOMiBN»eMOO)»#ooooooooo 

a«'tt»AH.Nnaoi<oan«NNooaiii.«a.^iii«haa«aaa«na«ooooooooo 

>>nnOhto<ONh-aannaoNnin«eNaaaaa«aaaiaoiai««8ianaooooooooo 

^Mnn'raaaar^r.r.aaaaaaaaaaaaaaaaaaaaaaaaaaaaooooooooo 


8i*>»a»a»'-i9a»*r»a^a^^a  —  aw«f>taa»t«— anw  —  «onmr>aaaaooooooooeo 
aanori  —  atP'-HaaaaaooacMwttamr.on'rwr.aaaaaaaaaaoooooooooo 

r.wn*>an'*r.maM«onaa~is«aah-h.aaaaaaaaaaaaaaaaaoooooooooo 

'NnntitatSNhsaavgiaaaaaoiaaaagaoioaaaannaacioooooooooo 


802299a?09S02990S0000000000000o000000°ooooooooooooo 
-MnTni^oao-Nn»ioor-oaO'-Mp)»in«3hooa'-N<>)<nn«r'a)(ao-NnTnef'ffl(ao 
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Table  Is  Probabilities  for  a  Biased  Gaussian  vs.  Fraction  of  Standard  Deviation 


TABLE  SHOWING  PER-CENT  OF  READINGS  TAKEN  FROM  A  GAUSSIAN  DISRTIBUTION  WITH  SIGMA*1  AND  THE  INDICATED  BIASES 

ERROR-BAR  -TO*  BIAS"  .00  .20  .40  .60  .80  1.00  1.20  1.40  1.60  1.80  2.00 

FRACTION  OF  R.M.S. 


ONO-lOQNNnOin-fflaia-NNIflNninNNIOOOlOQOOOOOOOOOOOOOOOOOOOOOO 

o*«9<DNco^*-«o<rcaf'*wr'or4CM5rcDr^<o<0coo)Q>0>0>ooQOOooooooooooooooooooo 


Niofflwaj«n**0(iiMfl''h-v(Ohffloio)ffloioifl)0)oooQoooooooooooooooooooo 

**^Nn^intniohO}oa)O)O)0)O)O>aiO)Oio)O)0iooQooooooooooooooQoooooo 


oO'-'N'-Nono  -  NOOif^i/)N\ninr^r*co<N®c»><NtDa>c»oooooooooQooooooooooooo 
owo)ooaji/)(j)0)»*No)fflao'-Nin'"N(ji7iofflO)(j)0)0)OQooooooooooooooooooooo 


n®».(0^BlflnNONVOl0OP)lflMDfflfll(Jl0IIJIfll0)0)OOOOOOOOOOOOOOOOOOOOOOO 

Hi-NNAVIOlOlOMSOOIOlOlOIOIOtOlQOOIOlOXAOQOOQOOOOOOOOOOOOOOOOOO 


o  cs  ,s  ^  r*.CNajcoou)iOr*io«~csr«ooaj»i  *-a»iDairo*-incoa>cnooooooooooooooooooooQ 
ONioinciQioonffl«Nion-o<Nionioinotii3(flO)aiflicjHJiooooooooooooooooooooo 


'Tajtooin  —  eototo^iD^Oino>rj^<or.a)0)0)OiO)0>cno)cnaiooooooooooooooooooooo  • 

■-■■MnnvuiisiiiMocoiogioioioinsiacijiaicioiaiiiiooooaooooaooooooooooQ 


OB«M»l»OlH)'Mmi0l(U’-ain'N»#IIIMll'-01<ir-lll01OOOOOOOOOQOOOOOOOOOQ 

o  —  vo<#wgw'-(~f'»i(D>0(DW  —  —  lomoiwio  —  ^ujcomoioiaiwoooooooooooooooooooo 


u>oa>  —  oo  »r  —  coinwco«»<i>*oo  —  min<or.«ooio>oioio»oio>oio>oooooooooooooooooooo 
—  —  NMNOOitoohMiaooioiaosaooiooiooAaoooooooooooooooooooo 


ooNMta'Mioin'-oinainonin'naNnoiooaiMviDisoiiiioooooooooooooooooo 
ooNo«oo«NMi«a«i<nnM><-  n  —  i»  —  «u>r*a>o>a>a>a>o>oooooooooooooooooo 


•  NaY'OO’.Nnoiooinr'aMoiOMingioioidiiiiouiioioigioooooooooooooooooo 
—  —  iwnn7U)in(0(Or'(^(D(oo>o)Oia>o>a>o>o>(»o>o>o)0)0)oio>o)oooooooooooooooooo 


ooratMnaiior.i'.oooinuxnoo  —  ini^<-Nin(oo>r''rn(oioO'T(0<oaio>oooooooooooooooo 
oo(Ou>mor.Mui(OC(inm(Oii)(a(Ooa>«r(oin(Mr.-«ior'oooicno)0)0)0)00oooooooooooooo 

(owof'^of'wcninouioiwioci  —  nmoNooooiaOKAOiaaaioicnoioooooooooooooooo 
—  NNn<i«ioiii(i)hNN(oaaoi(iiaioiaoioioaioaao9ioioioioioooooooooooooooo 


o  —  co  <t»  —  sonmooioiiiiooiiooniiiNiiinoovon-nNoiNioMooiaoooooooooooooo 
ovi'Ocs  —  oinmiiiifliiioi<nnN'tMaoo9ioii'-oisMoigaioioioioioioooooooooooooo 

r»ofwa»<or<o>i/j  —  co  —  in  a>  ra  u>  g>  —  ro<vici^r»ooooo>a>o»0>o>o><»<»0>o>0>a>oooooooooooooo 
—  (N<Nco'TTif>coior'r~r.ma>a>cncn(j>-.:0)<7iO!<»a)a>aia)a>cna)a>CT>cnc7>cnoooaooaooooooo 


Oioin^(n(Jiflinaior-ii)NQniiMDi(iMOitioiMii<nn'-n--r.-':«)^ojOioiooo(jooQooooo 

ot'«roo'rcjr.f~^<nMino(Oino  —  o>^ioiovomo»(N»»(Ot^ix>u>o>0)0>o)0><»o>oooooooooooo 

r.u>nor.q-O(Divr-(MU>on(0(»  —  rM7inu>r.a)ooooo>o>o>0)0>o>o>o>o>0)0>no>oooooooooooo 
—  <N(OPirruiin(OU)r-h.a)(atD(oo>oi(ji<J)0)OiO)0)aiO)Ci<7ioioiO)0)Oio)aioioio)Oooooooooooo 


OMiotnm  —  nonoonionaonoioisoioi.'  —  —  —  inr»»ouima)(^"Tu>r~(o<j>oiooooooooooo 
OOlh^O-OI'r'tO-MIMnMOOOOlNKl^MOlTO-nin'OMOalQfflOlOlOICnOlOODOOOOOOOO 


r'inc>>oa>,»  —  r.«ooN<oo(n®o>-M(Tui(Oh.r'(Dooo)0)<no)oia>o)<no)ai<no)0)0)aoooooooooo 
—  c*ow^'(ninto<oi~r~<Doo<Da)<j>o>o)o>c>a>o)<j>wo>woa>a)0>a>a>a)a>o>a>o)a>ooaoooooooo 


oouhbatoi''  r-.<i)<oa)w^^>»0>c'(<oin<,>N<oiocot»  —  nnn  -  oon«h«oooioooooooooQ 
ooioiooN'  in  ®  —  M<DO><oa>(£>oocoN»T'rMcoc'>f'Ocj'r<oi-~<DcDO)a>»a>cio>oa>oooooooooo 

—  t>««CS®O(<>®0>  —  M^(0(0r»r'00«0>0)0)«0>0)0>0)0>01010>010>0>0000000000 
—  N»n«iinin®®r'f'mromajo)cno)aio)<jio>o>oioio><noio)0io)0)0i(jici)oaio)0)oooooooooo 


OMACtaOM-  (OUlWNCOCOCOr*  —  01  W  W  —  00(010^0010)0)0000000000 

oocowon  —  coo  —  N<ooiiso«ooainooNii>nt'On<tiOMiiooioioiiiioioioioioQoooooooo 


mow  —  0(0  —  r*naM«on®o)  —  M«in®t>r.<oooo>o>oioioio>mo>o)oio>oioio)0>oooooooooo 
—  www(rinin®®r«r«oooo(oooo)oio>o>(oaio>o><no>o)0>o)o>o>o>o)0>o)oio>oio)o>oooooooooo 


ooooooooooooooooooooooooooooooooooooooooooooooooooo 
o  —  NnouiiDMOOio  -  Mw«ji/>ior--cooo  —  n  n  oi/i  10  r- o  oi  o  -  cscnvin®r.cocno  —  «  n  <nn  u  mo  o  o 

—  —  —  —  —  —  —  —  —  —  WNWNNWWCMNWWWWWWWWWWW'TV^^^r^'^’^^-^Irt 
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Table  2:  Probabilities  for  a  Biased  Gaussian  vs.  Fraction  of  Root  Mean  Square 


